--- title: Data Transforms for Hyperspectral Datacubes keywords: fastai sidebar: home_sidebar summary: "Hyperspectral datacubes contain three axes (cross-track, along-track, and wavelength) and are stored in 3D numpy ndarrays. When using a pushbroom scanner, the datacube are filled gradually. The implementation here is based on a circular buffer and we provide additional methods to apply a pipeline of transforms which, for example, can be used for smile correction, and radiance conversion. " description: "Hyperspectral datacubes contain three axes (cross-track, along-track, and wavelength) and are stored in 3D numpy ndarrays. When using a pushbroom scanner, the datacube are filled gradually. The implementation here is based on a circular buffer and we provide additional methods to apply a pipeline of transforms which, for example, can be used for smile correction, and radiance conversion. " nb_path: "00_data.ipynb" ---
{% raw %} 
/Users/eway/.pyenv/versions/3.8.3/lib/python3.8/site-packages/pandas/compat/__init__.py:97: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
  warnings.warn(msg)
{% endraw %} {% raw %}

class Array[source]

Array() :: ndarray

Use this to type-annotate numpy arrays, e.g. image: Array['H,W,3', np.uint8] xy_points: Array['N,2', float] nd_mask: Array['...', bool] from: https://stackoverflow.com/questions/35673895/type-hinting-annotation-pep-484-for-numpy-ndarray

{% endraw %} {% raw %}
{% endraw %}

Generic Circular Buffer on numpy.ndarrays

The base functionality is implemented on a generic circular buffer. The datatype dtype can be modified as desired but the default is set to store int32 digital numbers.

{% raw %}

class CircArrayBuffer[source]

CircArrayBuffer(size:tuple=(100, 100), axis:int=0, dtype:type=int32, show_func:Optional[ndarray]=None)

Circular FIFO Buffer implementation on ndarrays. Each put/get is a (n-1)darray.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

CircArrayBuffer.put[source]

CircArrayBuffer.put(line:ndarray)

Writes a (n-1)darray into the buffer

CircArrayBuffer.get[source]

CircArrayBuffer.get()

Reads the oldest (n-1)darray from the buffer

CircArrayBuffer.show[source]

CircArrayBuffer.show()

Display the data

{% endraw %}

For example, we can write to a 1D array

{% raw %} 
cib = CircArrayBuffer(size=(7,),axis=0)
for i in range(9):
    cib.put(i)
    cib.show()

for i in range(9):
    print(i,cib.get())

#(7) [0 0 0 0 0 0 0]
#(7) [0 1 0 0 0 0 0]
#(7) [0 1 2 0 0 0 0]
#(7) [0 1 2 3 0 0 0]
#(7) [0 1 2 3 4 0 0]
#(7) [0 1 2 3 4 5 0]
#(7) [0 1 2 3 4 5 6]
#(7) [7 1 2 3 4 5 6]
#(7) [7 8 2 3 4 5 6]
0 2
1 3
2 4
3 5
4 6
5 7
6 8
7 None
8 None
{% endraw %}

Or a 2D array

{% raw %} 
cib = CircArrayBuffer(size=(4,4),axis=0)
cib.put(1) # scalars are broadcasted to a 1D array
for i in range(5):
    cib.put(cib.get()+1)
    cib.show()
{% endraw %}

Loading Camera Settings and Calibration Files

The OpenHSI camera has a settings dictionary which contains these fields:

  • camera_id is your camera name,
  • row_slice indicates which rows are illuminated and we crop out the rest,
  • index2wavelength_range is a linear approximation of what the wavelength map is given by start, stop, step,
  • resolution is the full pixel resolution given by the camera without cropping, and
  • fwhm_nm specifies the size of spectral bands in nanometers,
  • exposure_ms is the camera exposure time last used,
  • luminance is the reference luminance to convert digital numbers to luminance,
  • longitude is the longitude degrees east,
  • latitude is the latitude degrees north,
  • datetime_str is the UTC time at time of data collection,
  • altitude is the altitude above sea level (assuming target is at sea level) measured in km,
  • radiosonde_station_num is the station number from http://weather.uwyo.edu/upperair/sounding.html,
  • radiosonde_region is the region code from http://weather.uwyo.edu/upperair/sounding.html, and
  • sixs_path is the path to the 6SV executable.

The pickle file is a dictionary with these fields:

  • camera_id is your camera name,
  • HgAr_pic is a picture of a mercury argon lamp's spectral lines for wavelength calibration,
  • flat_field_pic is a picture of a well lit for calculating the illuminated area,
  • smile_shifts is an array of pixel shifts needed to correct for smile error, and
  • rad_ref is a 4D datacube with coordinates of cross-track, wavelength, exposure, and luminance,
  • sfit is the spline fit function from the integrating sphere calibration, and
  • rad_fit is the interpolated function of the expected radiance at sensor computed using 6SV.

These files are unique to each OpenHSI camera.

{% raw %}

class CameraProperties[source]

CameraProperties(txt_path:str='assets/cam_settings.txt', pkl_path:str='assets/cam_calibration.pkl')

Save and load OpenHSI camera settings and calibration

{% endraw %} {% raw %}
{% endraw %} {% raw %}

CameraProperties.dump[source]

CameraProperties.dump()

Save the settings and calibration files

{% endraw %}

For example, the contents of CameraProperties consists of two dictionaries. To produce the files cam_settings.txt and cam_calibration.pkl, follow the steps outlined in the calibration module.

{% raw %} 
cam_prop = CameraProperties("assets/cam_settings.txt","assets/cam_calibration.pkl")
cam_prop

settings = 
{'camera_id': '0', 'row_slice': [197, 649], 'index2wavelength_range': [397.35239556600595, 823.5444046337864, 0.2085273916954825], 'resolution': [772, 2064], 'fwhm_nm': 4, 'exposure_ms': 10, 'longitude': -17.7, 'latitude': 146.1, 'datetime_str': '2021-05-26 03:26', 'altitude': 0.12, 'radiosonde_station_num': 94299, 'radiosonde_region': 'pac', 'sixs_path': 'assets/6SV1.1/sixsV1.1', 'luminance': 30000}

calibration = 
{'camera_id': '0', 'HgAr_pic': array([[32, 30, 30, ..., 31, 31, 32],
       [32, 32, 34, ..., 33, 32, 32],
       [32, 31, 30, ..., 32, 32, 31],
       ...,
       [31, 31, 31, ..., 33, 31, 33],
       [29, 32, 31, ..., 31, 31, 32],
       [32, 32, 32, ..., 31, 32, 33]], dtype=uint32), 'flat_field_pic': array([[33, 34, 33, ..., 34, 36, 33],
       [34, 33, 36, ..., 33, 33, 35],
       [33, 33, 32, ..., 33, 35, 32],
       ...,
       [33, 32, 34, ..., 36, 32, 37],
       [35, 33, 34, ..., 34, 33, 33],
       [33, 33, 33, ..., 31, 33, 36]], dtype=uint32), 'smile_shifts': array([ 9,  9, 10, 10, 10, 10, 10, 10,  9,  9,  9,  9,  9,  9,  9,  9,  9,
        9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,
        9,  9,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,
        8,  8,  8,  8,  8,  8,  8,  7,  8,  8,  8,  7,  7,  7,  7,  7,  7,
        7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,
        7,  7,  7,  7,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,
        6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  5,  5,
        5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,
        5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  4,  4,  4,  4,  4,  4,
        4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,
        4,  4,  4,  4,  4,  4,  4,  4,  3,  3,  3,  3,  3,  3,  3,  3,  3,
        3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,
        3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,
        3,  3,  3,  3,  3,  2,  3,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,
        2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,
        2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,
        2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,
        2,  2,  1,  1,  1,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,
        2,  1,  2,  2,  2,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  0,  0,  0,  1,  0,  1,  1,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0], dtype=int16), 'rad_ref': <xarray.DataArray 'datacube' (luminance: 13, exposure: 9, cross_track: 452, wavelength: 108)>
array([[[[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        ...,

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]]],


       [[[  634,   643,   652, ...,   630,   631,   631],
         [  636,   642,   648, ...,   640,   642,   643],
         [  635,   642,   648, ...,   847,   857,   857],
         ...,
         [  634,   640,   647, ...,  1072,  1051,  1031],
         [  633,   640,   647, ...,  1072,  1048,  1025],
         [  635,   642,   647, ...,  1067,  1042,  1021]],

        [[  639,   651,   659, ...,   636,   636,   635],
         [  641,   649,   656, ...,   647,   648,   650],
         [  641,   650,   657, ...,   899,   909,   908],
         ...,
         [  641,   649,   656, ...,  1171,  1148,  1120],
         [  639,   647,   656, ...,  1169,  1143,  1112],
         [  639,   650,   655, ...,  1162,  1133,  1110]],

        [[  645,   657,   668, ...,   639,   640,   640],
         [  645,   656,   664, ...,   651,   654,   656],
         [  647,   657,   666, ...,   951,   958,   961],
         ...,
         [  647,   656,   663, ...,  1267,  1237,  1209],
         [  645,   655,   663, ...,  1267,  1235,  1203],
         [  646,   657,   665, ...,  1259,  1221,  1197]],

        ...,

        [[  674,   694,   709, ...,   663,   664,   664],
         [  675,   693,   707, ...,   684,   686,   689],
         [  675,   692,   710, ...,  1208,  1221,  1222],
         ...,
         [  675,   690,   707, ...,  1750,  1699,  1651],
         [  673,   689,   703, ...,  1747,  1697,  1641],
         [  674,   692,   706, ...,  1732,  1674,  1629]],

        [[  672,   696,   718, ...,   659,   660,   659],
         [  672,   696,   715, ...,   684,   686,   691],
         [  674,   696,   717, ...,  1341,  1357,  1357],
         ...,
         [  676,   693,   714, ...,  2020,  1963,  1893],
         [  673,   695,   712, ...,  2021,  1957,  1888],
         [  674,   695,   715, ...,  1999,  1934,  1875]],

        [[  700,   730,   758, ...,   681,   682,   681],
         [  702,   730,   759, ...,   715,   719,   722],
         [  702,   732,   759, ...,  1598,  1612,  1613],
         ...,
         [  705,   730,   755, ...,  2513,  2427,  2343],
         [  701,   729,   753, ...,  2503,  2413,  2328],
         [  703,   731,   758, ...,  2472,  2383,  2310]]],


       [[[  675,   695,   714, ...,   661,   662,   662],
         [  676,   694,   712, ...,   682,   684,   688],
         [  676,   694,   713, ...,  1201,  1213,  1215],
         ...,
         [  677,   695,   711, ...,  1766,  1715,  1665],
         [  674,   693,   709, ...,  1764,  1711,  1657],
         [  678,   696,   711, ...,  1747,  1694,  1646]],

        [[  689,   712,   735, ...,   673,   674,   672],
         [  689,   713,   734, ...,   698,   697,   702],
         [  691,   713,   734, ...,  1327,  1340,  1342],
         ...,
         [  692,   710,   733, ...,  2009,  1951,  1886],
         [  689,   711,   730, ...,  2004,  1944,  1881],
         [  692,   713,   734, ...,  1985,  1921,  1866]],

        [[  705,   730,   757, ...,   684,   684,   683],
         [  705,   730,   755, ...,   712,   712,   719],
         [  703,   731,   756, ...,  1450,  1466,  1466],
         ...,
         [  706,   730,   753, ...,  2248,  2177,  2103],
         [  703,   728,   750, ...,  2245,  2172,  2097],
         [  704,   729,   753, ...,  2219,  2144,  2083]],

        ...,

        [[  754,   800,   846, ...,   721,   722,   720],
         [  756,   800,   846, ...,   769,   771,   780],
         [  756,   802,   847, ...,  2048,  2077,  2079],
         ...,
         [  762,   802,   844, ...,  3427,  3304,  3177],
         [  757,   801,   842, ...,  3423,  3301,  3172],
         [  759,   802,   848, ...,  3377,  3255,  3140]],

        [[  815,   873,   929, ...,   772,   772,   772],
         [  816,   873,   929, ...,   833,   838,   846],
         [  818,   875,   931, ...,  2435,  2467,  2476],
         ...,
         [  824,   877,   926, ...,  4167,  4016,  3860],
         [  818,   875,   925, ...,  4146,  3996,  3844],
         [  818,   876,   928, ...,  4105,  3945,  3802]],

        [[  886,   959,  1037, ...,   827,   830,   826],
         [  887,   959,  1033, ...,   910,   917,   924],
         [  890,   965,  1040, ...,  3046,  3089,  3102],
         ...,
         [  894,   966,  1039, ...,  5367,  5166,  4944],
         [  891,   964,  1034, ...,  5344,  5147,  4941],
         [  891,   965,  1037, ...,  5280,  5076,  4880]]],


       ...,


       [[[ 1256,  1455,  1665, ...,  1136,  1134,  1130],
         [ 1250,  1454,  1664, ...,  1631,  1672,  1719],
         [ 1266,  1465,  1677, ...,  8453,  8388,  8278],
         ...,
         [ 1289,  1485,  1684, ..., 11375, 10883, 10387],
         [ 1281,  1480,  1676, ..., 11299, 10849, 10367],
         [ 1277,  1478,  1681, ..., 11172, 10719, 10247]],

        [[ 1384,  1625,  1878, ...,  1240,  1242,  1237],
         [ 1383,  1625,  1878, ...,  1845,  1905,  1956],
         [ 1398,  1635,  1895, ..., 10098, 10027,  9877],
         ...,
         [ 1427,  1656,  1904, ..., 13615, 13037, 12443],
         [ 1416,  1659,  1897, ..., 13539, 12982, 12398],
         [ 1411,  1648,  1898, ..., 13371, 12818, 12253]],

        [[ 1514,  1795,  2094, ...,  1349,  1352,  1342],
         [ 1516,  1795,  2095, ...,  2072,  2137,  2201],
         [ 1532,  1815,  2113, ..., 11763, 11677, 11493],
         ...,
         [ 1561,  1840,  2122, ..., 15835, 15165, 14456],
         [ 1551,  1828,  2111, ..., 15744, 15096, 14428],
         [ 1551,  1830,  2114, ..., 15565, 14911, 14263]],

        ...,

        [[ 2173,  2645,  3167, ...,  1890,  1895,  1878],
         [ 2171,  2650,  3162, ...,  3178,  3299,  3410],
         [ 2206,  2688,  3193, ..., 20087, 19940, 19583],
         ...,
         [ 2255,  2730,  3213, ..., 27052, 25897, 24684],
         [ 2239,  2716,  3187, ..., 26888, 25788, 24605],
         [ 2229,  2701,  3198, ..., 26573, 25451, 24327]],

        [[ 2540,  3141,  3784, ...,  2204,  2202,  2185],
         [ 2540,  3146,  3784, ...,  3822,  3966,  4105],
         [ 2581,  3189,  3829, ..., 25052, 24869, 24404],
         ...,
         [ 2642,  3246,  3854, ..., 33746, 32304, 30756],
         [ 2627,  3226,  3822, ..., 33531, 32155, 30688],
         [ 2618,  3209,  3822, ..., 33171, 31786, 30335]],

        [[ 3197,  3990,  4855, ...,  2745,  2745,  2732],
         [ 3196,  3993,  4868, ...,  4929,  5130,  5311],
         [ 3254,  4052,  4913, ..., 33405, 33127, 32527],
         ...,
         [ 3332,  4126,  4946, ..., 44948, 43032, 40924],
         [ 3310,  4100,  4908, ..., 44677, 42827, 40861],
         [ 3300,  4086,  4905, ..., 44152, 42297, 40396]]],


       [[[ 1274,  1486,  1710, ...,  1198,  1203,  1193],
         [ 1274,  1485,  1707, ...,  2650,  2771,  2892],
         [ 1286,  1505,  1725, ..., 10606, 10364, 10031],
         ...,
         [ 1309,  1516,  1731, ..., 11979, 11464, 10934],
         [ 1302,  1516,  1722, ..., 11860, 11374, 10884],
         [ 1299,  1507,  1724, ..., 11778, 11286, 10799]],

        [[ 1408,  1657,  1932, ...,  1318,  1321,  1316],
         [ 1412,  1663,  1933, ...,  3076,  3217,  3358],
         [ 1428,  1684,  1960, ..., 12679, 12391, 11986],
         ...,
         [ 1455,  1704,  1960, ..., 14322, 13717, 13090],
         [ 1446,  1696,  1949, ..., 14186, 13629, 13012],
         [ 1442,  1694,  1954, ..., 14118, 13514, 12949]],

        [[ 1548,  1837,  2157, ...,  1437,  1448,  1439],
         [ 1546,  1843,  2153, ...,  3522,  3681,  3861],
         [ 1564,  1863,  2179, ..., 14776, 14427, 13949],
         ...,
         [ 1594,  1888,  2185, ..., 16692, 15974, 15225],
         [ 1583,  1880,  2175, ..., 16524, 15851, 15137],
         [ 1578,  1876,  2177, ..., 16414, 15734, 15045]],

        ...,

        [[ 2201,  2700,  3244, ...,  2036,  2044,  2031],
         [ 2211,  2709,  3256, ...,  5703,  5976,  6273],
         [ 2244,  2751,  3296, ..., 25196, 24613, 23755],
         ...,
         [ 2289,  2790,  3308, ..., 28509, 27254, 25982],
         [ 2278,  2780,  3280, ..., 28224, 27078, 25846],
         [ 2266,  2765,  3285, ..., 28041, 26854, 25663]],

        [[ 2611,  3231,  3921, ...,  2397,  2410,  2402],
         [ 2611,  3247,  3926, ...,  6998,  7352,  7718],
         [ 2660,  3288,  3974, ..., 31439, 30713, 29651],
         ...,
         [ 2716,  3342,  3986, ..., 35596, 34055, 32438],
         [ 2695,  3323,  3954, ..., 35260, 33792, 32276],
         [ 2687,  3307,  3960, ..., 35023, 33521, 32036]],

        [[ 3297,  4123,  5031, ...,  3012,  3022,  3010],
         [ 3300,  4142,  5042, ...,  9153,  9600, 10092],
         [ 3354,  4198,  5108, ..., 41824, 40861, 39439],
         ...,
         [ 3432,  4266,  5133, ..., 47402, 45369, 43180],
         [ 3408,  4245,  5090, ..., 46984, 45022, 42994],
         [ 3393,  4218,  5089, ..., 46585, 44634, 42654]]],


       [[[  610,   612,   612, ...,   612,   611,   613],
         [  612,   612,   610, ...,   612,   612,   613],
         [  612,   611,   611, ...,   609,   612,   611],
         ...,
         [  612,   611,   611, ...,   612,   613,   611],
         [  611,   611,   611, ...,   613,   612,   607],
         [  613,   613,   611, ...,   614,   612,   612]],

        [[  610,   612,   612, ...,   612,   611,   612],
         [  612,   611,   609, ...,   614,   614,   615],
         [  612,   611,   611, ...,   610,   613,   612],
         ...,
         [  612,   611,   611, ...,   612,   614,   612],
         [  611,   610,   612, ...,   614,   614,   608],
         [  614,   614,   612, ...,   615,   613,   613]],

        [[  610,   612,   612, ...,   612,   610,   612],
         [  614,   612,   611, ...,   613,   615,   615],
         [  612,   612,   611, ...,   609,   613,   613],
         ...,
         [  612,   612,   611, ...,   613,   614,   612],
         [  614,   612,   612, ...,   615,   614,   609],
         [  614,   613,   612, ...,   615,   613,   612]],

        ...,

        [[  595,   596,   596, ...,   596,   595,   596],
         [  595,   596,   595, ...,   597,   597,   598],
         [  596,   596,   594, ...,   595,   597,   598],
         ...,
         [  596,   595,   596, ...,   596,   599,   596],
         [  596,   595,   596, ...,   598,   597,   591],
         [  598,   598,   596, ...,   600,   598,   597]],

        [[  595,   598,   597, ...,   596,   595,   597],
         [  597,   597,   596, ...,   599,   599,   600],
         [  597,   598,   596, ...,   594,   598,   597],
         ...,
         [  598,   596,   596, ...,   597,   600,   597],
         [  597,   597,   597, ...,   599,   600,   593],
         [  597,   598,   597, ...,   601,   597,   597]],

        [[  597,   600,   599, ...,   599,   599,   600],
         [  598,   598,   596, ...,   599,   599,   600],
         [  597,   599,   596, ...,   595,   600,   598],
         ...,
         [  600,   597,   599, ...,   598,   602,   599],
         [  599,   599,   598, ...,   602,   601,   596],
         [  599,   599,   597, ...,   602,   599,   598]]]], dtype=int32)
Coordinates:
  * cross_track  (cross_track) int32 0 1 2 3 4 5 6 ... 446 447 448 449 450 451
  * wavelength   (wavelength) float32 397.3 401.3 405.2 ... 813.3 817.3 821.3
  * luminance    (luminance) int32 0 2500 5000 7500 ... 45000 47500 50000 0
  * exposure     (exposure) int32 5 6 7 8 9 10 12 15 20, 'sfit': <scipy.interpolate.interpolate.interp1d object at 0x128d579a0>, 'rad_fit': <scipy.interpolate.interpolate.interp1d object at 0x128e57d60>}
{% endraw %} {% raw %} 
cam_prop.calibration["rad_ref"]
Show/Hide data repr Show/Hide attributes
xarray.DataArray
'datacube'
  • luminance: 13
  • exposure: 9
  • cross_track: 452
  • wavelength: 108
  • 0 0 0 0 0 0 0 0 0 0 0 ... 598 599 600 598 598 600 598 598 602 599 598
    array([[[[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        ...,

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]]],


       [[[  634,   643,   652, ...,   630,   631,   631],
         [  636,   642,   648, ...,   640,   642,   643],
         [  635,   642,   648, ...,   847,   857,   857],
         ...,
         [  634,   640,   647, ...,  1072,  1051,  1031],
         [  633,   640,   647, ...,  1072,  1048,  1025],
         [  635,   642,   647, ...,  1067,  1042,  1021]],

        [[  639,   651,   659, ...,   636,   636,   635],
         [  641,   649,   656, ...,   647,   648,   650],
         [  641,   650,   657, ...,   899,   909,   908],
         ...,
         [  641,   649,   656, ...,  1171,  1148,  1120],
         [  639,   647,   656, ...,  1169,  1143,  1112],
         [  639,   650,   655, ...,  1162,  1133,  1110]],

        [[  645,   657,   668, ...,   639,   640,   640],
         [  645,   656,   664, ...,   651,   654,   656],
         [  647,   657,   666, ...,   951,   958,   961],
         ...,
         [  647,   656,   663, ...,  1267,  1237,  1209],
         [  645,   655,   663, ...,  1267,  1235,  1203],
         [  646,   657,   665, ...,  1259,  1221,  1197]],

        ...,

        [[  674,   694,   709, ...,   663,   664,   664],
         [  675,   693,   707, ...,   684,   686,   689],
         [  675,   692,   710, ...,  1208,  1221,  1222],
         ...,
         [  675,   690,   707, ...,  1750,  1699,  1651],
         [  673,   689,   703, ...,  1747,  1697,  1641],
         [  674,   692,   706, ...,  1732,  1674,  1629]],

        [[  672,   696,   718, ...,   659,   660,   659],
         [  672,   696,   715, ...,   684,   686,   691],
         [  674,   696,   717, ...,  1341,  1357,  1357],
         ...,
         [  676,   693,   714, ...,  2020,  1963,  1893],
         [  673,   695,   712, ...,  2021,  1957,  1888],
         [  674,   695,   715, ...,  1999,  1934,  1875]],

        [[  700,   730,   758, ...,   681,   682,   681],
         [  702,   730,   759, ...,   715,   719,   722],
         [  702,   732,   759, ...,  1598,  1612,  1613],
         ...,
         [  705,   730,   755, ...,  2513,  2427,  2343],
         [  701,   729,   753, ...,  2503,  2413,  2328],
         [  703,   731,   758, ...,  2472,  2383,  2310]]],


       [[[  675,   695,   714, ...,   661,   662,   662],
         [  676,   694,   712, ...,   682,   684,   688],
         [  676,   694,   713, ...,  1201,  1213,  1215],
         ...,
         [  677,   695,   711, ...,  1766,  1715,  1665],
         [  674,   693,   709, ...,  1764,  1711,  1657],
         [  678,   696,   711, ...,  1747,  1694,  1646]],

        [[  689,   712,   735, ...,   673,   674,   672],
         [  689,   713,   734, ...,   698,   697,   702],
         [  691,   713,   734, ...,  1327,  1340,  1342],
         ...,
         [  692,   710,   733, ...,  2009,  1951,  1886],
         [  689,   711,   730, ...,  2004,  1944,  1881],
         [  692,   713,   734, ...,  1985,  1921,  1866]],

        [[  705,   730,   757, ...,   684,   684,   683],
         [  705,   730,   755, ...,   712,   712,   719],
         [  703,   731,   756, ...,  1450,  1466,  1466],
         ...,
         [  706,   730,   753, ...,  2248,  2177,  2103],
         [  703,   728,   750, ...,  2245,  2172,  2097],
         [  704,   729,   753, ...,  2219,  2144,  2083]],

        ...,

        [[  754,   800,   846, ...,   721,   722,   720],
         [  756,   800,   846, ...,   769,   771,   780],
         [  756,   802,   847, ...,  2048,  2077,  2079],
         ...,
         [  762,   802,   844, ...,  3427,  3304,  3177],
         [  757,   801,   842, ...,  3423,  3301,  3172],
         [  759,   802,   848, ...,  3377,  3255,  3140]],

        [[  815,   873,   929, ...,   772,   772,   772],
         [  816,   873,   929, ...,   833,   838,   846],
         [  818,   875,   931, ...,  2435,  2467,  2476],
         ...,
         [  824,   877,   926, ...,  4167,  4016,  3860],
         [  818,   875,   925, ...,  4146,  3996,  3844],
         [  818,   876,   928, ...,  4105,  3945,  3802]],

        [[  886,   959,  1037, ...,   827,   830,   826],
         [  887,   959,  1033, ...,   910,   917,   924],
         [  890,   965,  1040, ...,  3046,  3089,  3102],
         ...,
         [  894,   966,  1039, ...,  5367,  5166,  4944],
         [  891,   964,  1034, ...,  5344,  5147,  4941],
         [  891,   965,  1037, ...,  5280,  5076,  4880]]],


       ...,


       [[[ 1256,  1455,  1665, ...,  1136,  1134,  1130],
         [ 1250,  1454,  1664, ...,  1631,  1672,  1719],
         [ 1266,  1465,  1677, ...,  8453,  8388,  8278],
         ...,
         [ 1289,  1485,  1684, ..., 11375, 10883, 10387],
         [ 1281,  1480,  1676, ..., 11299, 10849, 10367],
         [ 1277,  1478,  1681, ..., 11172, 10719, 10247]],

        [[ 1384,  1625,  1878, ...,  1240,  1242,  1237],
         [ 1383,  1625,  1878, ...,  1845,  1905,  1956],
         [ 1398,  1635,  1895, ..., 10098, 10027,  9877],
         ...,
         [ 1427,  1656,  1904, ..., 13615, 13037, 12443],
         [ 1416,  1659,  1897, ..., 13539, 12982, 12398],
         [ 1411,  1648,  1898, ..., 13371, 12818, 12253]],

        [[ 1514,  1795,  2094, ...,  1349,  1352,  1342],
         [ 1516,  1795,  2095, ...,  2072,  2137,  2201],
         [ 1532,  1815,  2113, ..., 11763, 11677, 11493],
         ...,
         [ 1561,  1840,  2122, ..., 15835, 15165, 14456],
         [ 1551,  1828,  2111, ..., 15744, 15096, 14428],
         [ 1551,  1830,  2114, ..., 15565, 14911, 14263]],

        ...,

        [[ 2173,  2645,  3167, ...,  1890,  1895,  1878],
         [ 2171,  2650,  3162, ...,  3178,  3299,  3410],
         [ 2206,  2688,  3193, ..., 20087, 19940, 19583],
         ...,
         [ 2255,  2730,  3213, ..., 27052, 25897, 24684],
         [ 2239,  2716,  3187, ..., 26888, 25788, 24605],
         [ 2229,  2701,  3198, ..., 26573, 25451, 24327]],

        [[ 2540,  3141,  3784, ...,  2204,  2202,  2185],
         [ 2540,  3146,  3784, ...,  3822,  3966,  4105],
         [ 2581,  3189,  3829, ..., 25052, 24869, 24404],
         ...,
         [ 2642,  3246,  3854, ..., 33746, 32304, 30756],
         [ 2627,  3226,  3822, ..., 33531, 32155, 30688],
         [ 2618,  3209,  3822, ..., 33171, 31786, 30335]],

        [[ 3197,  3990,  4855, ...,  2745,  2745,  2732],
         [ 3196,  3993,  4868, ...,  4929,  5130,  5311],
         [ 3254,  4052,  4913, ..., 33405, 33127, 32527],
         ...,
         [ 3332,  4126,  4946, ..., 44948, 43032, 40924],
         [ 3310,  4100,  4908, ..., 44677, 42827, 40861],
         [ 3300,  4086,  4905, ..., 44152, 42297, 40396]]],


       [[[ 1274,  1486,  1710, ...,  1198,  1203,  1193],
         [ 1274,  1485,  1707, ...,  2650,  2771,  2892],
         [ 1286,  1505,  1725, ..., 10606, 10364, 10031],
         ...,
         [ 1309,  1516,  1731, ..., 11979, 11464, 10934],
         [ 1302,  1516,  1722, ..., 11860, 11374, 10884],
         [ 1299,  1507,  1724, ..., 11778, 11286, 10799]],

        [[ 1408,  1657,  1932, ...,  1318,  1321,  1316],
         [ 1412,  1663,  1933, ...,  3076,  3217,  3358],
         [ 1428,  1684,  1960, ..., 12679, 12391, 11986],
         ...,
         [ 1455,  1704,  1960, ..., 14322, 13717, 13090],
         [ 1446,  1696,  1949, ..., 14186, 13629, 13012],
         [ 1442,  1694,  1954, ..., 14118, 13514, 12949]],

        [[ 1548,  1837,  2157, ...,  1437,  1448,  1439],
         [ 1546,  1843,  2153, ...,  3522,  3681,  3861],
         [ 1564,  1863,  2179, ..., 14776, 14427, 13949],
         ...,
         [ 1594,  1888,  2185, ..., 16692, 15974, 15225],
         [ 1583,  1880,  2175, ..., 16524, 15851, 15137],
         [ 1578,  1876,  2177, ..., 16414, 15734, 15045]],

        ...,

        [[ 2201,  2700,  3244, ...,  2036,  2044,  2031],
         [ 2211,  2709,  3256, ...,  5703,  5976,  6273],
         [ 2244,  2751,  3296, ..., 25196, 24613, 23755],
         ...,
         [ 2289,  2790,  3308, ..., 28509, 27254, 25982],
         [ 2278,  2780,  3280, ..., 28224, 27078, 25846],
         [ 2266,  2765,  3285, ..., 28041, 26854, 25663]],

        [[ 2611,  3231,  3921, ...,  2397,  2410,  2402],
         [ 2611,  3247,  3926, ...,  6998,  7352,  7718],
         [ 2660,  3288,  3974, ..., 31439, 30713, 29651],
         ...,
         [ 2716,  3342,  3986, ..., 35596, 34055, 32438],
         [ 2695,  3323,  3954, ..., 35260, 33792, 32276],
         [ 2687,  3307,  3960, ..., 35023, 33521, 32036]],

        [[ 3297,  4123,  5031, ...,  3012,  3022,  3010],
         [ 3300,  4142,  5042, ...,  9153,  9600, 10092],
         [ 3354,  4198,  5108, ..., 41824, 40861, 39439],
         ...,
         [ 3432,  4266,  5133, ..., 47402, 45369, 43180],
         [ 3408,  4245,  5090, ..., 46984, 45022, 42994],
         [ 3393,  4218,  5089, ..., 46585, 44634, 42654]]],


       [[[  610,   612,   612, ...,   612,   611,   613],
         [  612,   612,   610, ...,   612,   612,   613],
         [  612,   611,   611, ...,   609,   612,   611],
         ...,
         [  612,   611,   611, ...,   612,   613,   611],
         [  611,   611,   611, ...,   613,   612,   607],
         [  613,   613,   611, ...,   614,   612,   612]],

        [[  610,   612,   612, ...,   612,   611,   612],
         [  612,   611,   609, ...,   614,   614,   615],
         [  612,   611,   611, ...,   610,   613,   612],
         ...,
         [  612,   611,   611, ...,   612,   614,   612],
         [  611,   610,   612, ...,   614,   614,   608],
         [  614,   614,   612, ...,   615,   613,   613]],

        [[  610,   612,   612, ...,   612,   610,   612],
         [  614,   612,   611, ...,   613,   615,   615],
         [  612,   612,   611, ...,   609,   613,   613],
         ...,
         [  612,   612,   611, ...,   613,   614,   612],
         [  614,   612,   612, ...,   615,   614,   609],
         [  614,   613,   612, ...,   615,   613,   612]],

        ...,

        [[  595,   596,   596, ...,   596,   595,   596],
         [  595,   596,   595, ...,   597,   597,   598],
         [  596,   596,   594, ...,   595,   597,   598],
         ...,
         [  596,   595,   596, ...,   596,   599,   596],
         [  596,   595,   596, ...,   598,   597,   591],
         [  598,   598,   596, ...,   600,   598,   597]],

        [[  595,   598,   597, ...,   596,   595,   597],
         [  597,   597,   596, ...,   599,   599,   600],
         [  597,   598,   596, ...,   594,   598,   597],
         ...,
         [  598,   596,   596, ...,   597,   600,   597],
         [  597,   597,   597, ...,   599,   600,   593],
         [  597,   598,   597, ...,   601,   597,   597]],

        [[  597,   600,   599, ...,   599,   599,   600],
         [  598,   598,   596, ...,   599,   599,   600],
         [  597,   599,   596, ...,   595,   600,   598],
         ...,
         [  600,   597,   599, ...,   598,   602,   599],
         [  599,   599,   598, ...,   602,   601,   596],
         [  599,   599,   597, ...,   602,   599,   598]]]], dtype=int32)
    • cross_track
      (cross_track)
      int32
      0 1 2 3 4 5 ... 447 448 449 450 451
      array([  0,   1,   2, ..., 449, 450, 451], dtype=int32)
    • wavelength
      (wavelength)
      float32
      397.3 401.3 405.2 ... 817.3 821.3
      units :
      nm
      array([397.3, 401.3, 405.2, 409.2, 413.2, 417.1, 421.1, 425. , 429. , 433. ,
       436.9, 440.9, 444.9, 448.8, 452.8, 456.7, 460.7, 464.7, 468.6, 472.6,
       476.6, 480.5, 484.5, 488.4, 492.4, 496.4, 500.3, 504.3, 508.3, 512.2,
       516.2, 520.1, 524.1, 528.1, 532. , 536. , 539.9, 543.9, 547.9, 551.8,
       555.8, 559.8, 563.7, 567.7, 571.6, 575.6, 579.6, 583.5, 587.5, 591.5,
       595.4, 599.4, 603.3, 607.3, 611.3, 615.2, 619.2, 623.1, 627.1, 631.1,
       635. , 639. , 643. , 646.9, 650.9, 654.8, 658.8, 662.8, 666.7, 670.7,
       674.7, 678.6, 682.6, 686.5, 690.5, 694.5, 698.4, 702.4, 706.4, 710.3,
       714.3, 718.2, 722.2, 726.2, 730.1, 734.1, 738. , 742. , 746. , 749.9,
       753.9, 757.9, 761.8, 765.8, 769.7, 773.7, 777.7, 781.6, 785.6, 789.6,
       793.5, 797.5, 801.4, 805.4, 809.4, 813.3, 817.3, 821.3], dtype=float32)
    • luminance
      (luminance)
      int32
      0 2500 5000 7500 ... 47500 50000 0
      units :
      Cd/m^2
      array([    0,  2500,  5000,  7500, 10000, 20000, 30000, 40000, 42500, 45000,
       47500, 50000,     0], dtype=int32)
    • exposure
      (exposure)
      int32
      5 6 7 8 9 10 12 15 20
      units :
      ms
      array([ 5,  6,  7,  8,  9, 10, 12, 15, 20], dtype=int32)
{% endraw %}

Transforms

We can apply a number of transforms to the camera's raw data and these tranforms are used to modify the processing level during data collection. For example, we can perform a fast smile correction and wavelength binning during operation. With more processing, this is easily extended to obtain radiance and reflectance.

Some transforms require some setup which is done using CameraProperties.tfm_setup. This method also allows one to tack on an additional setup function with the argument more_setup which takes in any callable which can mutate the CameraProperties class.

{% raw %}

CameraProperties.tfm_setup[source]

CameraProperties.tfm_setup(more_setup:Optional[CameraProperties]=None, dtype:Union[int32, float32]=int32)

Setup for transforms

{% endraw %} {% raw %}
{% endraw %} {% raw %}

CameraProperties.crop[source]

CameraProperties.crop(x:ndarray)

Crops to illuminated area

{% endraw %} {% raw %}

CameraProperties.fast_smile[source]

CameraProperties.fast_smile(x:ndarray)

Apply the fast smile correction procedure

{% endraw %} {% raw %}

CameraProperties.fast_bin[source]

CameraProperties.fast_bin(x:ndarray)

Changes the view of the datacube so that everything that needs to be binned is in the last axis. The last axis is then binned.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

CameraProperties.dn2rad[source]

CameraProperties.dn2rad(x:Array'>[ForwardRef('λ,x'), int32])

Converts digital numbers to radiance (uW/cm^2/sr/nm). Use after cropping to useable area.

{% endraw %} {% raw %}

CameraProperties.rad2ref_6SV[source]

CameraProperties.rad2ref_6SV(x:Array'>[ForwardRef('λ,x'), float32])

{% endraw %} {% raw %}
{% endraw %} {% raw %}

CameraProperties.set_processing_lvl[source]

CameraProperties.set_processing_lvl(lvl:int=2, custom_tfms:List[Callable[ndarray, ndarray]]=None)

Define the output of the transform pipeline. 0 : raw digital numbers cropped to useable sensor area 1 : case 0 + fast smile correction 2 : case 1 + fast binning (default) 3 : case 2 + conversion to radiance in units of uW/cm^2/sr/nm 4 : case 3 except radiance conversion moved to 2nd step 5 : case 3 + conversion to reflectance 6 : smile corrected and binned -> radiance 7 : case 6 + converted to reflectance

{% endraw %} {% raw %}
{% endraw %}

You can add your own tranform by monkey patching the CameraProperties class.

@patch
def identity(self:CameraProperties, x:np.ndarray) -> np.ndarray:
    """The identity tranform"""
    return x

If you don't require any camera settings or calibration files, a valid transform can be any Callable that takens in a 2D np.ndarray and returns a 2D np.ndarray.

Pipeline for Composing Transforms

Depending on the level of processing that one wants to do real-time, a number of transforms need to be composed in sequential order. To make this easy to customise, you can use the pipeline method and pass in a raw camera frame and an ordered list of transforms.

To make the transforms pipeline easy to use and customise, you can use the CameraProperties.set_processing_lvl method.

{% raw %}

CameraProperties.pipeline[source]

CameraProperties.pipeline(x:ndarray)

Compose a list of transforms and apply to x.

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
cam_prop = CameraProperties("assets/cam_settings.txt","assets/cam_calibration.pkl")

cam_prop.set_processing_lvl(-1) # raw digital numbers
test_eq( (772,2064), np.shape( cam_prop.pipeline(cam_prop.calibration["HgAr_pic"])) )

cam_prop.set_processing_lvl(0) # cropped
test_eq( (452,2064), np.shape( cam_prop.pipeline(cam_prop.calibration["HgAr_pic"])) )

cam_prop.set_processing_lvl(1) # fast smile corrected
test_eq( (452,2054), np.shape( cam_prop.pipeline(cam_prop.calibration["HgAr_pic"])) )

cam_prop.set_processing_lvl(2) # fast binned
test_eq( (452,108),  np.shape( cam_prop.pipeline(cam_prop.calibration["HgAr_pic"])) )

cam_prop.set_processing_lvl(3) # radiance
test_eq( (452,108),  np.shape( cam_prop.pipeline(cam_prop.calibration["HgAr_pic"])) )

cam_prop.set_processing_lvl(5) # reflectance
test_eq( (452,108),  np.shape( cam_prop.pipeline(cam_prop.calibration["HgAr_pic"])) )

cam_prop.set_processing_lvl(4) # radiance conversion moved earlier in pipeline
test_eq( (452,108),  np.shape( cam_prop.pipeline(cam_prop.calibration["HgAr_pic"])) )
{% endraw %}

Buffer for Data Collection

DataCube takes a line with coordinates of wavelength (x-axis) against cross-track (y-axis), and stores the smile corrected version in its CircArrayBuffer.

For facilitate near real-timing processing, a fast smile correction procedure is used. An option to use a fast binning procedure is also available. When using these two procedures, the overhead is roughly 2 ms on a Jetson board.

Instead of preallocating another buffer for another data collect, one can use the circular nature of the DataCube and use the internal buffer again without modification - just use DataCube.push like normal.

Storage Allocation

All data buffers are preallocated so it's no secret that hyperspectral datacubes are memory hungry. For reference:

along-track pixels wavelength binning RAM needed time to collect at 10 ms exposure time to save to SSD
4096 4 nm ≈ 800 MB ≈ 55 s ≈ 3 s
1024 no binning ≈ 4 GB ≈ 14 s ≈ 15 s

In reality, it is very difficult to work with raw data without binning due to massive RAM usage and extended times to save the NetCDF file to disk which hinders making real-time analysis. The frame rate (at 10 s exposure) with binning drops the frame rate to from 90 fps to 75 fps. In our experimentation, using a SSD mounted into a M.2 slot on a Jetson board provided the fastest experience. When using other development boards such as a Raspberry Pi 4, the USB 3.0 port is recommended over the USB 2.0 port.

{% raw %}

class DateTimeBuffer[source]

DateTimeBuffer(n:int=16)

Records timestamps in UTC time.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

DateTimeBuffer.update[source]

DateTimeBuffer.update()

Stores current UTC time in an internal buffer when this method is called.

{% endraw %} {% raw %} 
timestamps = DateTimeBuffer(n=8)
for i in range(8):
    timestamps.update()
    
timestamps.data

array([datetime.datetime(2021, 10, 3, 3, 6, 8, 338152, tzinfo=datetime.timezone.utc),
       datetime.datetime(2021, 10, 3, 3, 6, 8, 338200, tzinfo=datetime.timezone.utc),
       datetime.datetime(2021, 10, 3, 3, 6, 8, 338208, tzinfo=datetime.timezone.utc),
       datetime.datetime(2021, 10, 3, 3, 6, 8, 338214, tzinfo=datetime.timezone.utc),
       datetime.datetime(2021, 10, 3, 3, 6, 8, 338220, tzinfo=datetime.timezone.utc),
       datetime.datetime(2021, 10, 3, 3, 6, 8, 338226, tzinfo=datetime.timezone.utc),
       datetime.datetime(2021, 10, 3, 3, 6, 8, 338232, tzinfo=datetime.timezone.utc),
       datetime.datetime(2021, 10, 3, 3, 6, 8, 338238, tzinfo=datetime.timezone.utc)],
      dtype=object)
{% endraw %} {% raw %}

class DataCube[source]

DataCube(n_lines:int=16, processing_lvl:int=2, txt_path:str='assets/cam_settings.txt', pkl_path:str='assets/cam_calibration.pkl') :: CameraProperties

docstring.

{% endraw %} {% raw %}
{% endraw %} {% raw %}

DataCube.put[source]

DataCube.put(x:ndarray)

Applies the composed tranforms and writes the 2D array into the data cube. Stores a timestamp for each push.

{% endraw %} {% raw %}

DataCube.save[source]

DataCube.save(save_dir:str)

Saves to a NetCDF file to directory dir_path in folder given by date with file name given by UTC time.

{% endraw %} {% raw %}
{% endraw %}

For example, if this data was collected on 2021/10/03 at 12:18:30.010568 UTC, then there will be two files saved under the directory 2021_10_03. The two files are the NetCDF file 12_18_30.nc and PNG image 12_18_30.png.

{% raw %}

DataCube.load_nc[source]

DataCube.load_nc(nc_path:str)

Lazy load a NetCDF datacube into the DataCube buffer.

{% endraw %} {% raw %}
{% endraw %}

The plot_lib argument hs Choose matplotlib if you want to use Choose bokeh if you want to compose plots together and use interactive tools.

{% raw %}

DataCube.show[source]

DataCube.show(plot_lib:str='bokeh', red_nm:float=640.0, green_nm:float=550.0, blue_nm:float=470.0, robust:bool=False, hist_eq:bool=False)

Generate a histogram equalised RGB plot from chosen RGB wavelengths. The plotting backend can be specified by plot_lib and can be "bokeh" or "matplotlib".

{% endraw %} {% raw %}
{% endraw %} {% raw %} 
n = 256

dc = DataCube(n_lines=n,processing_lvl=2,txt_path="assets/cam_settings.txt",pkl_path="assets/cam_calibration.pkl")

for i in range(200):
    dc.put( np.random.randint(0,255,dc.settings["resolution"]) )

dc.show("bokeh")
{% endraw %} {% raw %} 
dc.save("assets")

<ipython-input-370-eb3c3214b5ac>:15: DeprecationWarning: parsing timezone aware datetimes is deprecated; this will raise an error in the future
  time=(["time"],self.timestamps.data.astype(np.datetime64))), attrs={})
{% endraw %} {% raw %} 
dc.load_nc(Path("assets/").ls(file_exts="")[0].ls(file_exts=".nc")[0])

---------------------------------------------------------------------------
NotADirectoryError                        Traceback (most recent call last)
<ipython-input-409-839e0c0ec2d5> in <module>
----> 1 dc.load_nc(Path("assets/").ls(file_exts="")[1].ls(file_exts=".nc")[0])

~/.pyenv/versions/3.8.3/lib/python3.8/site-packages/fastcore/xtras.py in ls(self, n_max, file_type, file_exts)
    233 def save_pickle(fn, o):
    234     "Save a pickle file, to a file name or opened file"
--> 235     with open_file(fn, 'wb') as f: pickle.dump(o, f)
    236 
    237 # Cell

~/.pyenv/versions/3.8.3/lib/python3.8/site-packages/fastcore/foundation.py in __call__(cls, x, *args, **kwargs)
     95     def __call__(cls, x=None, *args, **kwargs):
     96         if not args and not kwargs and x is not None and isinstance(x,cls): return x
---> 97         return super().__call__(x, *args, **kwargs)
     98 
     99 # Cell

~/.pyenv/versions/3.8.3/lib/python3.8/site-packages/fastcore/foundation.py in __init__(self, items, use_list, match, *rest)
    103     def __init__(self, items=None, *rest, use_list=False, match=None):
    104         if (use_list is not None) or not is_array(items):
--> 105             items = listify(items, *rest, use_list=use_list, match=match)
    106         super().__init__(items)
    107 

~/.pyenv/versions/3.8.3/lib/python3.8/site-packages/fastcore/basics.py in listify(o, use_list, match, *rest)
     54     elif isinstance(o, list): res = o
     55     elif isinstance(o, str) or is_array(o): res = [o]
---> 56     elif is_iter(o): res = list(o)
     57     else: res = [o]
     58     if match is not None:

~/.pyenv/versions/3.8.3/lib/python3.8/site-packages/fastcore/xtras.py in <genexpr>(.0)
    231 
    232 # Cell
--> 233 def save_pickle(fn, o):
    234     "Save a pickle file, to a file name or opened file"
    235     with open_file(fn, 'wb') as f: pickle.dump(o, f)

~/.pyenv/versions/3.8.3/lib/python3.8/pathlib.py in iterdir(self)
   1116         if self._closed:
   1117             self._raise_closed()
-> 1118         for name in self._accessor.listdir(self):
   1119             if name in {'.', '..'}:
   1120                 # Yielding a path object for these makes little sense

NotADirectoryError: [Errno 20] Not a directory: 'assets/.DS_Store'
{% endraw %} {% raw %} 
dc.show("bokeh")
{% endraw %}